Scanning environments and tracking unmanned aerial vehicles

ABSTRACT

Systems and methods for scanning environments and tracking unmanned aerial vehicles within the scanned environments are disclosed. A method in accordance with a particular embodiment includes using a rangefinder off-board an unmanned air vehicle (UAV) to identify points in a region. The method can further include forming a computer-based map of the region with the points and using the rangefinder and a camera to locate the UAV as it moves in the region. The location of the UAV can be compared with locations on the computer-based map and, based upon the comparison, the method can include transmitting guidance information to the UAV. In a further particular embodiment, two-dimensional imaging data is used in addition to the rangefinder data to provide color information to points in the region.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/127,476, filed on Mar. 3, 2015, titled SCANNING ENVIRONMENTS ANDTRACKING UNMANNED AERIAL VEHICLES, which is herein incorporated byreference in its entirety.

TECHNICAL FIELD

The present disclosure is directed generally to scanning environmentsand tracking unmanned aerial vehicles.

BACKGROUND

Unmanned aerial vehicles (UAVs) have become increasingly popular devicesfor carrying out a wide variety of tasks that would otherwise beperformed with manned aircraft or satellites. Such tasks includesurveillance tasks, imaging tasks, and payload delivery tasks. However,UAVs have a number of drawbacks. For example, it can be difficult tooperate UAVs, particularly autonomously, in close quarters, e.g., nearbuildings, trees, or other objects. In particular, it can be difficultto prevent the UAVs from colliding with such objects. While techniquesexist for controlling UAVs and other autonomous vehicles in complexenvironments, such techniques typically rely on hardware that is tooheavy, too bulky, and/or requires too much power to be carried by atypical UAV without dramatically impacting the range and/or payloadcapacity of the UAV. Accordingly, there remains a need for techniquesand associated systems that can allow UAVs to safely and accuratelynavigate within their working environments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1C are partially schematic, block diagrams illustrating anoverall process for scanning an environment and tracking a UAV withinthe environment, in accordance with an embodiment of the presenttechnology.

FIG. 2 is a flow diagram of a representative scanning and trackingprocess configured in accordance with a particular embodiment of thepresent technology.

FIG. 3 illustrates a representative scanner configured in accordancewith an embodiment of the present technology.

FIG. 4 is a block diagram illustrating components of a representativescanner operating in a scanning mode, in accordance with arepresentative embodiment of the present technology.

FIG. 5 is a block diagram illustrating components of a representativescanner operating in a tracking mode, in accordance with an embodimentof the present technology.

FIGS. 6A and 6B are schematic illustrations a UAV having a codingarrangement to facilitate identifying and/or tracking the UAV inaccordance with an embodiment of the present technology.

FIG. 7 illustrates an arrangement of multiple scanners positioned toscan an environment in accordance with an embodiment of the presenttechnology.

FIG. 8 illustrates an arrangement for carrying out a scanning operationvia equipment carried by a UAV in accordance with an embodiment of thepresent technology.

FIG. 9A schematically illustrates a conventional process for scanning anobject.

FIG. 9B schematically illustrates a technique for selectively scanningfeatures of an object in accordance with an embodiment of the presenttechnology.

FIG. 10 is a schematic block diagram illustrating a process foridentifying and scheduling features to be scanned in accordance with anembodiment of the present technology.

FIG. 11 is a flow diagram illustrating a process for planning andexecuting a flight path trajectory in accordance with an embodiment ofthe present technology.

FIG. 12 illustrates a scanner positioned in an environment in accordancewith an embodiment of the present technology.

FIGS. 13A-13B illustrate isometric and plan views, respectively, of animage of the environment shown in FIG. 12.

FIG. 14 illustrates an image of another portion of the environment shownin FIG. 12.

FIGS. 15A-15C illustrate front, top, and side views of an environmentthat includes wind turbine, and an associated UAV flight path, inaccordance with an embodiment of the present technology.

FIGS. 16A-16D illustrate a process for obtaining data from wind turbineblades while the blades are in motion, in accordance an embodiment ofthe present technology.

DETAILED DESCRIPTION

The present technology is directed generally to unmanned aerial vehicles(UAVs) and more particularly, to systems and methods for scanning theenvironments in which unmanned aerial vehicles operate, planning aflight for the unmanned aerial vehicles through one or more of theenvironments, and tracking the motion of the unmanned aerial vehicles inthose environments. Specific details of several embodiments of thedisclosed technology are described below with reference to particular,representative configurations. In other embodiments, the disclosedtechnology may be practiced in accordance with UAVs and associatedsystems having other configurations. In still further embodiments,particular aspects of the disclosed technology may be practiced in thecontext of autonomous vehicles other than UAVs (e.g., autonomous cars orwatercraft). Specific details describing structures or processes thatare well-known and often associated with UAVs, but that mayunnecessarily obscure some significant aspects of the presentlydisclosed technology, are not set forth in the following description forpurposes of clarity. Moreover, although the following disclosure setsforth several embodiments of different aspects of the disclosedtechnology, several other embodiments of the technology can haveconfigurations and/or components different than those described in thissection. As such, the present technology may have other embodiments withadditional elements and/or without several of the elements describedbelow with reference to FIGS. 1A-15C.

Several embodiments of the disclosed technology may take the form ofcomputer-executable instructions, including routines executed by aprogrammable computer or controller. Those skilled in the relevant artwill appreciate that the technology can be practiced on computer orcontroller systems other than those shown and described below. Thetechnology can be embodied in a special-purpose computer, controller, ordata processor that is specifically programmed, configured orconstructed to perform one or more of the computer-executableinstructions described below. Accordingly, the terms “computer” and“controller” as generally used herein include a suitable data processorand can include internet appliances and hand-held devices, includingpalm-top computers, wearable computers, cellular or mobile phones,multi-processor systems, processor-based or programmable consumerelectronics, network computers, laptop computers, mini-computers and thelike. Information handled by these computers can be presented at anysuitable display medium, including a liquid crystal display (LCD). As isknown in the art, these computers and controllers commonly have variousprocessors, memories (e.g., non-transitory computer-readable media),input/output devices, etc.

The present technology can also be practiced in distributedenvironments, where tasks or modules are performed by remote processingdevices that are linked through a communications network. In adistributed computing environment, program modules or subroutines may belocated in local and remote memory storage devices. Aspects of thetechnology described below may be stored or distributed oncomputer-readable media, including magnetic or optically readable orremovable computer discs, as well as distributed electronically overnetworks. Data structures and transmissions of data particular toaspects of the technology are also encompassed within the scope of thepresent technology.

1. Overview

FIGS. 1A-1C schematically identify several functions that can beperformed as part of the technology disclosed in the presentapplication. FIG. 1A illustrates a process for scanning an environment190 in which a UAV is expected to operate. The environment 190 can beindoors and/or outdoors, and include one or more objects 191, forexample, a building. A system 100 in accordance with an embodiment ofthe present technology includes a scanner 110 that is used to scan theenvironment 190 and, in particular, the object or objects 191, toproduce a computer-based model 136 of the environment 190. The scanner110 can be controlled by a controller 130. The same controller 130 canbe used to perform a variety of additional functions, also describedherein, in accordance with particular embodiments of the presenttechnology. In other embodiments, these functions may be divided overmultiple controllers. Multiple controllers may be referred to herein as“controllers,” and one or more controllers can be included in a “controlsystem.” As discussed above, the controller 130 can have any of avariety of suitable configurations, e.g., a laptop computerconfiguration.

As shown schematically in FIG. 1B, the controller 130 can include aprocessor 131, a memory 132 (e.g., non-transitory computer-readablemedia), and one or more input/output devices 133. The input/outputdevices 133 can include an input device 134 (e.g., a keyboard) and anoutput device 135 (e.g., a screen displaying a graphical userinterface). As shown in FIG. 1B, the output device 135 is displaying acomputer-based object image 136 of the environment 190, including theobject 191 shown in FIG. 1A. The output device 135 also displays aproposed flight path image 137, representing a proposed flight path fora UAV in the region of the object 191. Accordingly, the controller 130can be used to plan the UAV's flight path, e.g., by identifyingobstacles in the flight path, alerting the operator to such obstacles,and/or suggesting or automatically implementing alternate flight paths.

A further aspect of the system is illustrated in FIG. 1 C. Thecontroller 130 and the scanner 110, initially described above withreference to FIG. 1A, can be used to track the motion of a UAV 150 as itoperates in the environment 190 and around the object 191. Inparticular, the scanner 110 can be used in this context to identify theactual location of the UAV 150 relative to the computer-based model ofthe object 191, and can communicate this information to the UAV 150 viaa communication device 138 (e.g., an antenna). Accordingly, the system100 can be used to provide real-time data regarding the UAV's positionand, by comparing that data to the model described above with referenceto FIGS. 1A and 1 B, can provide positional feedback to the UAV. Thefeedback can be used to adjust the UAV's actual flight path 151, asnecessary, in comparison to the planned flight path 137 to allow the UAVto carry out its mission with a high degree of precision and/or avoidcollisions.

Thus, the system 100 can be used to (a) map the environment in which theUAV operates, (b) facilitate defining a flight plan for the UAV in theenvironment, and (c) provide feedback to the UAV if it deviates from theflight plan. Further details of each of these functions are describedbelow with reference to FIGS. 2-15C.

2. Scanning and Tracking

FIG. 2 is a flow diagram illustrating a representative process forscanning an environment and tracking a UAV within the environment, inaccordance with an embodiment of the present technology. The overallprocess can include the subprocesses of scanning, planning a flightpath, and tracking the UAV. Depending on the embodiment, thesesubprocesses can be performed sequentially as part of a single overallprocess, or any of the subprocesses can be performed individually. Theprocess 200 can include scanning the local environment with a ground orother off-board scanner (block 201). In a representative embodiment, thescanner is located on the ground, e.g., supported by a tripod or anothersuitable structure. In other embodiments, the scanner can have otherlocations, e.g., it can be carried aloft by a balloon or aircraft.

In block 202, the process includes planning a flight path within theenvironment, where the flight path may be manually input by a humanoperator or calculated automatically based on various constraints. Indecision block 203, the process includes determining whether the plannedpath is achievable, for example, determining if the path is free ofobstacles. If not, the process returns to block 202 for an update to theplanned flight path. In various embodiments, the process can excludedetected obstacles when computing a revised path. If the path isachievable (e.g., is free of obstacles given the proximity constraintsand dynamics of the UAV), the process proceeds to block 204, whichbegins the tracking portion of the process. As part of this process, thesystem acquires a track on the UAV as it is in use, e.g., as it takesoff and flies. When the tracking information is acquired (e.g., when thelocation of the UAV is acquired with a camera and/or rangefinder), thislocation information can be transmitted to the controller 130 thatcontrols the tracking device in pan and tilt axes, continuously keepingthe UAV within the field of view. The controller 130 launches the UAV(block 205) and directs the UAV to the beginning of its planned flightpath (block 206). In block 207, the UAV follows the flight path to itsconclusion, and at block 208 lands at the end of the mission. During thecourse of the flight, the information received via tracking the UAV(e.g., the coordinates of the UAV) can be transmitted to the controller130 to update the trajectory of the UAV. For example, if the UAV beginsdeviating from the pre-defined flight plan, the information can be usedto guide (e.g., automatically) the UAV back to the planned route.Accordingly, portions of the controller can be located on-board the UAV,off-board the UAV, or both, depending on the manner in which UAV controland feedback are implemented.

As the UAV flies (blocks 205-208), the process includes checking, e.g.,continually checking, to determine whether the UAV is successfully beingtracked (block 209). As long as the UAV is being successfully tracked,the process continues as indicated. If the system loses track of theUAV, then in block 210, the process includes taking actions tofacilitate re-acquiring the track on the UAV. Such steps can includedirecting the UAV to hover, land, and/or fly toward the scanner, and/orperforming a sweeping (and/or other) motion with the scanner to locatethe UAV, and/or other suitable actions.

As part of the process of flying the UAV, the operator can manipulatethe progress of the UAV along the flight path. For example, the operatorcan use controls similar to VCR controls to fast-forward, reverse, orpause the progress of the UAV along the flight path. Such controls, ofcourse, will be dependent on the UAV configuration, e.g., the UAV musthave a hover capability in order for it to be paused. A similar approachcan be used to preview the flight of the UAV along a proposed flightpath, as part of the planning process for the flight path, which isdescribed further below with reference to FIGS. 11-15C.

FIG. 3 is an isometric illustration of a representative scanner 110having features configured in accordance with an embodiment of thepresent technology. The scanner 110 can include a support 111 thatcarries a camera 117 and a rangefinder 118. The camera 117 can beconfigured to produce two-dimensional optical images of the environmentaround the scanner 110 by receiving radiation from the environment inthe visible spectrum, infrared range, and/or other suitable frequencyranges. The rangefinder 118 can include an emitter 119 and a receiver120. The emitter 119 emits a signal that reflects from an object in theenvironment and is received by the receiver 120. The distance from thescanner 110 to the object is then determined or estimated by using anyof a variety of suitable techniques, including estimating the amount oftime required for the signal to transit from the emitter 119 to theobject and back to the receiver 120 (“time of flight”). Accordingly, thecamera 117 can identify and transmit two-dimensional information aboutthe environment, and the rangefinder 118 can add the third dimension.

The camera 117 and the rangefinder 118 can be carried by a tilt stage114 and can be moved together as a unit to scan the environment aroundthe scanner 110. The tilt stage 114 carries a tilt motor 115 thatrotates the camera 117 and the rangefinder 118, as a unit, about a firstaxis (e.g., a horizontal axis H). A corresponding tilt encoder 116tracks the motion of the camera 117 and the rangefinder 118 relative tothe horizontal axis H. A pan motor 112 carried by the support 111rotates the tilt stage 114 (including the camera 117 and the rangefinder118) as a unit about a second axis (e.g., a vertical axis V). A panencoder 113 tracks the rotational position of the camera 117 and therangefinder 118 around the vertical axis V. Accordingly, the pan motor112 and the tilt motor 115 can rotate to the camera 117 and therangefinder 118 through arcs sufficient to cover a roughly hemisphericalvolume around the scanner 110.

In a particular embodiment, the rangefinder 118 can include a lidardetector, which emits and receives laser light (e.g., IR laser light).Suitable lidar detectors have range capabilities in the hundreds ofmeters, depending on factors that include the size of the emitter 119and receiver or detector 120, and the ranging technology used. Asdiscussed above the ranging technology can include a time of flighttechnique in one embodiment. In other embodiments, other techniques,such as SETS techniques, can produce suitable results without requiringdirect time of flight calculations, at lower cost and lower (but stillsuitable) resolution. The scans can be conducted in a methodical sweeppattern, or a coarse scan followed by a detailed scan, or an adaptivescan (described further below with reference to FIGS. 9A-10), or viaanother suitable technique. In other embodiments, the rangefinder 118can emit signals other than a laser signal, suitable for detecting thedistance between the scanner 110 and the objects around it. For example,a radar signal can be used for tracking, though it is expected that alaser signal will out-perform a radar signal for scanning. In anotherembodiment, the laser scanner can be replaced by multiplehigh-resolution cameras or a structured light arrangement to perform thescanning process.

FIG. 4 is a schematic block diagram illustrating the system 100operating in a scanning mode, in accordance with an embodiment of thepresent technology. As shown in FIG. 4, the camera 117 records images ofthe surrounding environment 190, while the rangefinder 118 records thedistances to the objects within the environment 190. The camera 117accordingly generates RGB data (or other optical data) 123 and therangefinder 118 generates depth or distance data 124. The RGB data 123is transmitted to the processor 131. The depth data 124 is convertedfrom spherical coordinates to Cartesian coordinates at the processorusing conventional converter logic 125 that operates on information fromthe tilt encoder 116 and the pan encoder 113. This transformedcoordinate information is used to generate a point cloud from the datacaptured by the camera 117 and the rangefinder 118 using the point cloudbuilder logic 127.

The system 100 can also include a motor control unit that providesinstructions to the tilt motor 115 and the pan motor 112, and is itselfunder the control of a scan controller 430. The scan controller 430 caninclude scan region logic 122, which is described in further detailbelow with reference to FIGS. 9A-10. The scan controller 430 can be astand-alone controller, or it can be integrated with one or more othercontrollers, e.g., the controller 130 described above with reference toFIGS. 1A-1C.

A power supply 126 provides power to the various components of thesystem 100. The input/output device 133 receives information from anoperator and provides output information to the operator. The result ofthe process performed during the scanning mode shown in FIG. 4 is acomputer-based model (e.g., a point cloud) of the environment 190.

In particular embodiments, the data obtained during the scanningoperation and used to build the three-dimensional model can besupplemented with additional data. For example, the model can beaugmented or enhanced with photographs or other sensor readings taken bythe scanner camera 117 or by a UAV that flies through the scannedenvironment. This operation can be conducted in real time in someembodiments, and offline in others. The enhancement can include addingthe color information contained in the camera image to the points in thepoint cloud to produce a more realistic, colored model displaying thespectral representation at each point.

The data obtained from a particular scan can be stored and used later bythe same scanner (e.g., in a track mode), or by a different scanner,also in the track mode, or by a UAV operating independently of anoffboard tracking device. In a particular embodiment, when a tracker ispositioned in a particular area, it can automatically access prior scansmade of that environment, and those scans can be downloaded to thescanner as it operates in the track mode. Furthermore, a particular scanmay be used at multiple locations, not necessarily the same as thelocation at which the initial scan was made. For example, a scan made ofa particular wind turbine, power line tower, or other structure can beused to survey or inspect other structures that have an identicalconfiguration, for example, wind turbines of the same make and model. Insuch cases, the scanned information can be calibrated to the particularlocation of an individual wind turbine, which may be different for eachwind turbine.

FIG. 5 illustrates the system 100 operating in a tracking mode. In thismode, the camera 117 identifies a UAV 150 (e.g. by tracking a fiducialsuch as an LED carried by the UAV, by matching airframe characteristicsas trained by a machine learning algorithm, or by other means) andgenerates two-dimensional tracking data 123. Simultaneously, therangefinder 118 generates range data 561 corresponding to the distancebetween the scanner 110 and the UAV 150, as the UAV flies. Thisinformation is provided to the processor 131 which generates an estimateof the position of the UAV 150 using position estimator logic 562. Theposition information can be transmitted to the I/O device 133, which cantransmit the information via an information signal 563 to the UAV 150.With this information, the UAV 150 can modify, adjust, and/or compensatefor variations in the flight path it executes. The position informationis also transmitted to the Motor Control Unit 121 which actuates the panmotor 112 and tilt motor 115 to continuously keep the UAV 150 within thefield of view of the camera 117 and in front of the rangefinder 118 asthe UAV flies.

In another embodiment, the system can track the UAV 150 by combiningcorner cube interferometry with a laser-based position sensing device togenerate the position estimate for the UAV 150. The position informationcan be transmitted to the I/O device 133 and UAV 150 so that it canmodify, adjust, and/or compensate for variations in its flight path.

In particular embodiments, the system can, in addition to transmittinginformation to the UAV 150, receive information from the UAV 150. In aparticular aspect of this embodiment, the information received from theUAV 150 can correspond to an identity of the UAV 150 and/or anorientation of the UAV 150. FIGS. 6A and 6B illustrate a representativeUAV 650 having a body 652 and one or more rotors 653 (four are shown inFIGS. 6A and 6B) for propulsion. The UAV 650 can also include a physicalencoding schema, for example a ring 654 or lighting system that in turnincludes one or more coding patterns 655 (FIG. 6B). The coding patterns655 can be used to identify the type of UAV detected by the scanner 110(FIG. 5) during its tracking process, for example, to confirm that theproper UAV is being tracked. This procedure can also be used to identifynew or additional UAVs that may enter the environment in which thetracking process is being conducted. Accordingly, the coding patterns655 can perform the function of an identifying fiducial.

In addition, the coding patterns 655 can have different characteristicsat different circumferential positions around the coding ring 654.Accordingly, when the system 100 detects the coding patterns 655, it candetect the orientation of the UAV 650. This information can in turn beused to help control, guide, and/or direct the UAV 650. To obtain thisinformation, the camera 117 (or a separate, dedicated camera) can obtainan image of the coding patterns, which is then processed (e.g., in themanner of a bar coder). In another embodiment, the system 100 can issuean inquiry signal 664 (e.g., in addition to the information signal 563)that impinges on and is reflected by the coding pattern 655. Thereflected signal is then processed to determine the orientation and/oridentity of the UAV 650. In both of the foregoing embodiments, thecoding pattern 655 is passive. In other embodiments, the coding pattern655 can be actively transmitted by the UAV 650, e.g., via a radioemission, an LED pattern or flash sequence, and/or other suitablearrangements.

An advantage of the orientation function described above is that it maybe more accurate than the orientation function provided by an on-boardinertial navigation system. Another advantage is that on-board inertialnavigation systems typically include a magnetometer, which can beaffected by nearby structures, e.g., magnetic or metal structures. Theoff-board scanner can provide orientation information that is unaffectedby such structures, allowing the UAV to operate in environments it mightnot otherwise be able to.

In at least some instances, a single scanner may be insufficient toprovide all the data required for the UAV to carry out its tasks. Forexample, as shown in FIG. 7, a target object 791 may include a buildinghaving target features that are not accessible to a single scanner at asingle location. In one embodiment, the scanner 110 is moved amongmultiple locations to provide sufficient data for the desired pointcloud. At each location, the scanner 110 obtains two-dimensional imagedata and depth data, as discussed above, and the multiple data sets canbe “stitched” together to form a composite computer model. In otherembodiments, multiple scanners 110 can be used to both scan the featuresof the building, and provide tracking for the UAV 150 as it flies itsmission. The scanners 110 can communicate with each other, as indicatedby arrows M so as to hand off tracking responsibility from one scannerto the other as the UAV 150 conducts its mission.

In yet another embodiment, illustrated in FIG. 8, the scanning functioncan be performed by a UAV 850 itself. In such an embodiment, thescanning equipment is made compact and lightweight enough to be carriedby the UAV 850. In some embodiments, the scanning equipment may becompromised in some manner to reduce its weight. For example, thescanning equipment may have a coarser resolution, reduced range, reducedfield of view and/or other weight-saving features. However, with thesystem on board, the UAV 850 can assume the positions identified byletters A, B, C and D to map the features of the building 791. Aftermapping the features, the UAV 850 can carry out its mission, asindicated by letter E.

In the foregoing embodiment, the tracking function described above withreference to FIG. 5 may be performed by comparing images taken by theUAV 850 while it is in flight, with point cloud information produced byscans taken by the UAV while it is on the ground at positions A-D. Inanother embodiment, the tracking operation is carried out by two UAVs850, each equipped with a scanner. Accordingly, one UAV 850 can fly amission (e.g., an inspection mission) and the second UAV 850 can trackthe first UAV 850. The tracking function can be performed from any ofthe scanning positions A-D described above, or (with accommodations foran offset) from a different location.

The system can perform still further functions directed to tracking theUAVs. For example, in the track mode, the system can scan theenvironment to determine if a UAV is in the area. Upon detecting theUAV, the tracker can use any of the arrangements described above toidentify the UAV and, optionally, its orientation. The tracker can alsocommunicate with UAVs that are out of visual range or beyond line ofsight (BLOS). Such UAVs can relay their position and trajectory to thetracker (for example, via GPS and radio communication) and the trackercan accordingly orient itself toward the position where the UAV isexpected to come within range. Once the UAV comes within range, thetracker can carry out the tracking operations described above. In otherembodiments, the UAV can be tracked using ADS-B in addition to, or inlieu of GPS. Still further, the system can scan a pre-scannedenvironment to check for new objects that may have entered theenvironment after the initial scan was completed. Such new objects caninclude a UAV, or other objects that may represent obstacles to a UAV.If such objects are detected, the three-dimensional model of theenvironment can be updated to reflect the existence of those objects.

As noted above with reference to FIG. 7, some arrangements can includemultiple scanners for performing scanning and/or tracking functions. Insuch cases, these scanners can communicate with each other to providecontinuity within the three-dimensional models that they create, and totransmit information related to UAVs in their individual environments.For example, the scanners can cooperate to give control priority to thescanner that is closest to the UAV, or that is projected to have thelongest contact with the UAV before a handoff. In addition, whether anoperation includes one scanner or more than one scanner the system canschedule on-board photo/recording/sensing functions. For example, it maybe advantageous to direct the UAV to perform a sensing function from aprecise location. When the scanner is performing in the tracking mode,it can alert the UAV when the UAV reaches the precise location, in realtime, to ensure that the sensing function is performed at the properlocation. This technique can also be used to capture an image of the UAVwhen it is in a particular location, using the ground-based cameraportion of the scanner. In this manner, the UAV can capture an image ata particular location, and the scanner can capture an image of the UAVat that location. Both techniques can be used to improve thereproducibility of data, e.g., if the same image is to be obtained atdifferent times. The UAV position information from the tracker can beused to reduce computation and improve the fit of photos to the cloud ofpoints. For example, the downstream analysis of the data can be simplerand/or more accurate when the data are tagged with the precise locationfrom which the data were taken.

3. Adaptive Scanning

FIG. 9A illustrates an object 991 being scanned by a scanner 110 inaccordance with a conventional scanning technique. Using this technique,the scanner 110 methodically and sequentially pans and tilts to producescan paths 994 a (shown in dotted lines) that cover the object 991. Thisprocess can be time-consuming, particularly when only certain featuresof the object 991 may be of real interest to the end user. For example,the end user may be more interested in certain target features 992. Thetarget features 992 can include straight lines that form target regions993, for example, the roof and windows of the building.

FIG. 9B illustrates the scanner 110 carrying out an adaptive scanroutine in accordance with an embodiment of the present technology. Inthis embodiment, the scan paths 994 b executed by the scanner 110 aredirected specifically to the target features 992 and therefore thetarget regions 993. As a result, the scanner 110 spends less timeunnecessarily scanning features that are not of interest. Accordingly,the process of scanning the object 991 can take significantly less timethan it does using conventional techniques, and can use significantlyfewer resources.

FIG. 10 is a block diagram illustrating a process for carrying out thescanning operation shown in FIG. 9B. An overall process 1000 inaccordance with an embodiment of the present disclosure includes animage processing block 1001 that in turn includes an image acquisitionblock 1002. At block 1002, the process includes acquiring image data,e.g., from the camera 117. At block 1003, the image data are analyzed,for example to identify particular features, and more particularly,features that are likely to be of interest, e.g., by an end user. Suchfeatures can include lines, corners, and/or other target shapes, orfeatures that may be identified by a user.

At block 1004, the target shapes are grouped into regions, which maycorrespond to doors, windows, and/or other shapes or areas. Block 1005and 1006 include classifying each of the target regions, for example, byselecting regions that are of interest for targeted scanning by therangefinder 118 (block 1009). This process can be performed manually,automatically, or via a combination of manual and automated techniques.For example, the process can include using computer-based shaperecognition routines to identify target features (or target regions) anda user can approve or reject the automatically generated selections. Inother embodiments, the user can assume full control of the selectionprocess (fully manual) or cede control (fully automated). In particularembodiments, algorithms such as Hough transforms are used to identifyfeatures and/or regions of interest. In other embodiments, machinelearning and/or neural network techniques can be used to identifyfeatures and/or regions of interest, thus reducing the required userinput. If the features and/or regions are of interest, a scheduler 1007collects the information for each region and identifies an optimal (orat least expedient) scan path (block 1008). At block 1009, the scancontroller 430 (FIG. 4) directs the camera 117 and the associatedrangefinder 118 (FIG. 4) along the scan paths identified at block 1008.The output (block 1010) is a three-dimensional model, which the UAV usesto navigate and perform its mission.

4. Flight Path Planning

FIGS. 11-15C illustrate representative processes for forming or definingflight paths in accordance with which the UAV conducts its mission inthe environment scanned by the scanner, as described above. Arepresentative process 1100 shown in FIG. 11 includes scanning theenvironment or importing a three-dimensional model of the environment(block 1101). At block 1102, the process includes constructing a flightpath within the environment. In a particular embodiment, the flight pathis constructed using user input 1103. For example, the user can view thethree-dimensional model on a computer screen and, via a mouse, keys, ajoystick, and/or other input devices, construct the flight path. Becausethe flight path is three-dimensional, the system can allow the user torotate the image in any manner (e.g., tumble the image) so as to viewthe environment from multiple perspectives and provide a suitablethree-dimensional flight path.

In another embodiment, the flight path can be automatically generated,as indicated by block 1104. In this case, the process may requireadditional inputs 1105. Such inputs can include a generalcharacterization of the desired flight path, for example, anidentification of the regions of interest and the distance away from theregions of interest that the UAV is to fly. It may be of particularimportance to maintain a set distance from an object, (a) to preventcollisions with the object, and/or (b) to keep the object within thetarget range of a sensor aboard the UAV. Additionally, the informationcan include the speed of the UAV, the flight dynamics and capabilitiesof the UAV, and any restrictions or limitations associated with theability of the UAV to point its camera or other sensor toward the targetregion. Based on this information, and an input from the user (e.g., theuser can click on a feature or object, and select an offset distance),the system can automatically generate a flight path. The flight path canbe a single-pass path or a multi-pass path (e.g., a “lawnmower” pattern)depending on factors including the field of view of the UAV-basedsensor, and the desired resolution of the object it senses.

Block 1106 includes calculating a predicted trajectory based on the UAVpath determined at block 1102, taking into account the speed of the UAV,the flight dynamics and capabilities of the UAV, and other restrictionsas described above. Block 1107 includes checking to determine if thepath is free of collisions, for example, collisions with the objectsrepresented in the model imported at block 1101. If the path doesinclude collisions (or greater than a threshold likelihood ofcollisions), the process returns to block 1102 for adjustments to theflight path. As an example, the process can prevent areas of the paththat are causing the collisions from being included in a subsequentconstruction of a suitable path. If not, then the flight path can beinitiated and executed, as indicated at block 1108.

FIG. 12 illustrates the scanner 110 in a representative environment 1290that includes first objects (e.g., statues 1291 a) and second objects(e.g., buildings 1291 b). FIG. 13A illustrates the computer-based modelof the environment, including the statues 1291 a and the buildings 1291b, based on data obtained from the scanner 110. FIG. 13A alsoillustrates a proposed flight path 1337 that may be constructed by auser viewing the pointcloud of the environment 1336. FIG. 13B is a planview (top-down) of the pointcloud environment 1336 shown in FIG. 13A.The user can toggle between preset views (e.g., plan view, side view,front view, etc.), or can move gradually and incrementally at any pacefrom one view to another by tumbling around the pointcloud environment1336.

FIG. 14 illustrates one of the buildings 1291 b shown in FIGS. 13A and13B along with the proposed flight path 1337. The proposed flight path1337 includes an interfering portion 1337 a where the UAV would collidewith, or come too close to, the building 1291 b. Accordingly, the useris alerted to the interference, for example, by displaying theinterfering portion 1337 a of the flight path in a different color thanthe non-interfering portion. In other embodiments, the system can useother alerting techniques (e.g., flashing lights, and/or an audiblesignal). Additionally, the system can prevent the user from flying themission until the interfering portion is removed, e.g. by modifying theflight path.

FIGS. 15A, 15B, and 15C are front, top, and side views of an object 1591that includes a wind turbine 1591. FIGS. 15A-C also illustrate a flightpath 1551 along which a UAV 150 flies to examine the wind turbine 1591,and a (schematic) field of view 1593 for the ground-based scanner. Thethree-view illustration is suitable for indicating to the user that nocollisions or likely collisions exist for the UAV 150. As discussedabove, the image may also be tumbled or otherwise rotated to providethis information to the user. In any of the foregoing embodiments, thedata collected by the scanner can be used to aid the UAV in performingat least two functions. First, the data can be used to aid the UAV incarrying out the substance of its mission. For example, as shown inFIGS. 15A-C, the mission may include inspecting the blades of a windturbine. Accordingly, the image data can provide an accurate indicationof the location of the blades and/or other areas of interest that theUAV may focus on as part of its mission. Second, the data can aid theUAV in avoiding obstacles as it carries out its mission. For example,the image data can allow the UAV to avoid striking the pylon 1592 thatcarries the wind turbine blades.

In the embodiment shown in FIGS. 15A-15C, the wind turbine 1591 isstopped while the UAV 150 performs its inspection. In anotherembodiment, the UAV 150 can perform the inspection while the turbineblades are spinning, as illustrated in a representative embodiment shownin FIGS. 16A-16D.

Beginning with FIG. 16A, the wind turbine 1591 includes the pylon 1592described above, which carries three blades 1595. When the blades 1595are in motion, they sweep out a disk D. Each blade 1595 can includemultiple regions extending inwardly from the blade tips. Tworepresentative regions are illustrated in FIGS. 16A-16D as a firstregion 1596 a and a second region 1596 b. In the embodiment shown inFIG. 16A, the UAV 150 moves to a first position P1 located toward theouter periphery of the disk D, so as to be aligned with the firstregions 1596 a of each blade 1595 as the blades pass by.

Referring now to FIG. 16B, the first region 1596 a of one of the blades1595 has rotated into the field of view 1597 of a sensor carried by theUAV 150. With the first region 1596 a in the field of view 1597, asdetermined by the UAV and/or the ground-based scanner/tracker, thesensor is activated. For example, when the sensor includes a camera, thecamera is activated to take an image of the first region 1596 a. The UAV150 can remain in the first position P1 and, as the first region 1596 aof each successive blade 1595 passes into the field of view 1597, theUAV 150 can activate the sensor to take an image or collect other data.When the first regions 1596 a of each blade 1595 have been imaged orotherwise sensed, the UAV 150 can move to a second position P2, as shownin FIG. 16C. From the second position P2, the UAV 150 can image orotherwise sense the second regions 1596 b of each blade 1595, as shownin FIG. 16D. The foregoing process can be repeated for additionalregions of interest, e.g., regions located inwardly from the secondregion 1596 b toward the hub 1599 from which the blades 1595 extend. Anadvantage of the foregoing approach is that the wind turbine 1591 neednot be halted while the UAV 150 obtains diagnostic information,including but not limited to image data of the blades 1595. This in turncan increase the run time of the wind turbine 1591 without compromisingon the need for obtaining periodic diagnostic data.

One feature of at least some of the foregoing embodiments is that thesystem can include a combination of a camera (or other two-dimensionalimaging device) and a laser scanner (or other rangefinder) to map out anenvironment in which a UAV is expected to fly. One advantage of thisarrangement is that it is relatively inexpensive when compared totraditional surveying laser scanners, or spinning lidar devices. Anotheradvantage of the foregoing arrangement is that it provides capabilitiesdifferent than those of existing systems. For example, existingsurveying rangefinders operate at about 0.5-1 Hz and provide resolutionat a millimeter or sub-millimeter level. Embodiments of the presentsystem operate at rates of 50 Hz-100 Hz, which (unlike a 1 Hz rate) aresufficient to keep pace with the flight speeds and offset distances oftypical UAVs. The resolution of embodiments of the present system can bereduced to the centimeter level, which is sufficient for typical UAVtasks, and reduces the cost of the system.

Yet another expected advantage of embodiments of the disclosed systemsis that they can be located off-board the UAV and therefore do notrequire the UAV to provide payload capacity and processing power for thescanning and mapping equipment. Instead, the payload capacity can bereserved for mission-performance equipment, and the power can beconserved for conducting the UAV's mission. This feature can beparticularly important for UAVs that have limited battery capacity.Still another advantage is that the combined scanner and rangefinderarrangement is compact and is therefore more portable than other mappingequipment, for example, equipment that includes an array of infraredcameras.

Another feature of at least some of the foregoing embodiments is thatthe scanner can perform a tracking function, in addition to a scanningfunction. An advantage of this feature is that the UAV can be told whatits position is from an off-board source on roughly the same time scaleat which it is flying. In particular, the UAV can receive updatesmultiple times per second, and at a rate that allows it to change courseif necessary, for example, to avoid obstacles. Another advantage of thisfeature is that the UAV's position can be continually compared to apre-defined flight path that has been defined using an accurate,three-dimensional model of the local environment. Accordingly, thelikelihood for the UAV to get off track is reduced, and if the UAV doesget off track, the system can redirect the UAV with a high degree ofaccuracy.

Yet another feature of at least some of the foregoing embodiments isthat the process for planning a flight can be automated, semi-automated,and/or more visually interactive than conventional waypoint planningarrangements. An advantage of this arrangement is that it can speed upthe process for flightpath planning, particularly, in cluttered localenvironments. Another advantage is that the flight path can be moreaccurate, which again has particular utility in GPS-denied or crowdedareas where GPS is either unavailable or insufficient to provide thelevel of detail required to navigate through such areas. Yet anotheradvantage is that the interface can be intuitive. This is so for atleast the reason that the objects presented at the interface arevisually representative of the objects as they actually appear. Abuilding looks like a building, a tree looks like a tree, etc. Stillanother advantage is that the interface can better simulate (viarotation or tumbling) the actual three-dimensional environment, and theuser can appreciate the three-dimensional nature of the flightenvironment by rotating the image so as to see obstacles and targets ofinterest from multiple angles. This is unlike typical flight planningarrangements in which the user has to imagine the height of obstacles.Still another advantage is that at least some aspects of the flightplanning process can be automated. For example, the user can input arepresentative flight path, which can be automatically evaluated forcollision problems. The flight path can also automatically be adjustedto provide a constant offset from a particular object, for example, awind turbine blade that is being inspected by the UAV flying along theflight path.

Still another feature of at least some of the foregoing embodiments isthat the process for obtaining the data used to build thethree-dimensional model can be adapted to account for the environmentthat is scanned. In particular, the scanning process can be focused onareas of interest, rather than the environment as a whole. An advantageof this approach is that the process for scanning the environment can befaster because areas that are similar (e.g., large flat surfaces) andareas that are physically unscannable (e.g., the sky) may be avoided andtherefore do not take time to map. Another advantage is that the limitednumber of data points available for a particular model can be focused onthe areas of interest so that those areas are modeled with greaterresolution than areas that are of less importance. In addition, theoverall number of points required to define a particular model can bereduced. Still further, the demands on (and therefore the cost of) thescanning motors are reduced when fewer data points are taken. Thisapproach can reduce the cost/complexity of the overall system.

5. Further Embodiments

In particular embodiments, the scanner 110 remains in the same positionfor both the scanning operation and the tracking operation. In otherembodiments, the scanner may be moved between these two operations. Insuch cases, the process for tracking the UAV can include accounting foran offset between the position at which the scanner conducted the scan,and the position from which the scanner tracks the UAV. The offset canbe determined using GPS and/or other suitable techniques. A particularscan may be tagged with its GPS coordinates so that it can be used formultiple missions, each of which may have the tracker at a differentlocation. The GPS coordinates from which the initial scan is made canprovide a constant reference point suitable for multiple uses.Alternatively, a local coordinate system can be used in lieu of GPS.

In particular embodiments, the fiducial carried by the UAV can take theform of a coding pattern, as described above. In other embodiments, thefiducial can include other arrangements, for example, an LED or LEDpattern. In still further embodiments, when in the track mode, thescanner can identify the UAV by matching certain characteristics of theUAV (e.g., its silhouette) with a library of pre-identified silhouettesas trained by machine learning algorithms, e.g. a deep neural network.

From the foregoing, it will be appreciated that specific embodiments ofthe present technology have been described herein for purposes ofillustration, but that various modifications may be made withoutdeviating from the technology. For example, while certain embodimentshave been described in the context of UAVs, in other embodiments,similar techniques may be used for other types of robots, e.g.,water-based robots and in particular, submersible robots, which canconfront issues associated with three-dimensional environments similarto those confronted by air vehicles. In particular embodiments describedabove, the tracking mode includes tracking coordinates. In otherembodiments, the tracking mode can include tracking the direction oftravel, the UAV speed, and/or other variables, in addition to or in lieuof tracking the vehicle coordinates. The tracking process and theassociated scanning process can be performed outdoors, as shown inseveral of the Figures above, and can also be performed indoors. Thesystem can cycle or oscillate between the scanning mode and the trackingmode, even when used in the same environment, for example, to detectchanges in the environment. Such changes can then be imported into thepoint cloud or other computer-based image file so as to update theassociated three-dimensional model. Particular embodiments of the systemwere described above in the context of a quad-rotor UAV, and in otherembodiments, the UAV can have other quad-rotor or non-quad-rotorconfigurations.

The scanner can have different arrangements in different embodiments.For example, in a representative embodiment described above, the cameraand rangefinder each have independent optical paths to the environmentand the UAV. In another embodiment, a prism or other beam splitterprovides identical optical information to both the camera and therangefinder. This approach can improve the correspondence betweeninformation provided by the camera and information provided by therangefinder. In other embodiments, the scanner can include devices otherthan a single camera and a single rangefinder, e.g., multiple cameras(having different fields of view, wavelength sensitivities, and/or otherfunctional parameters) and/or multiple rangefinders (having differentsensitivities over different distances, and/or other functionalparameters).

Certain aspects of the technology described in the context of aparticular embodiment may be combined or eliminated in otherembodiments. For example, in particular embodiments, the flight pathplanning algorithm described above can be used independently of thescanning and tracking processes described above. In another embodiment,the adaptive scanning techniques described above can be usedindependently of the tracking and/or flight planning processes describedabove. Further, while advantages associated with certain embodiments ofthe technology have been described in the context of those embodiments,other embodiments may also exhibit such advantages, and not allembodiments need necessarily exhibit such advantages to fall within thescope of the present technology. Accordingly, the present disclosure andassociated technology can encompass other embodiments not expresslyshown or described herein.

The present application is related to the following co-pending,concurrently-filed applications, each of which is herein incorporated byreference in its entirety: U.S. patent application Ser. No. ______(Attorney Docket No. 115577-8001.US01), titled SCANNING ENVIRONMENTS ANDTRACKING UNMANNED AERIAL VEHICLES, U.S. patent application Ser. No.______ (Attorney Docket No. 115577-8003.US00), titled SCANNINGENVIRONMENTS AND TRACKING UNMANNED AERIAL VEHICLES, and PCT ApplicationNo. ______, (Attorney Docket No. 115577-8001.WO00), titled SCANNINGENVIRONMENTS AND TRACKING UNMANNED AERIAL VEHICLES. To the extent anymaterials incorporated herein by reference conflict with the presentdisclosure, the present disclosure controls.

1. A method, performed by a computing system having a memory and aprocessor, for planning a flight of an unmanned aerial vehicle in anenvironment using a three-dimensional model of the environment, themethod comprising: providing, for presentation to a user, informationcharacterizing a visual representation of the three-dimensional model ofthe environment, the environment including a plurality of objects; basedon an input from the user, shifting a viewpoint from which the userviews the visual representation of the three-dimensional model of theenvironment; receiving, from the user, input related to a proposedflight path for the unmanned aerial vehicle through the environment;calculating a predicted trajectory of the unmanned aerial vehiclethrough the environment based at least in part on the received input andflight dynamics and capabilities of the unmanned aerial vehicle; andchecking whether the calculated predicted trajectory of the unmannedaerial vehicle through the environment is free of collisions.
 2. Themethod of claim 1, further comprising: generating the three-dimensionalmodel of the environment at least part by scanning the environment witha camera and a rangefinder.
 3. The method of claim 1, furthercomprising: importing the three-dimensional model of the environment. 4.The method of claim 1, further comprising: generating thethree-dimensional model of the environment at least part by extractingspatial information from images captured by the unmanned aerial vehicleor another unmanned aerial vehicle.
 5. The method of claim 1, whereinthe received input related to the preferred flight path for the unmannedaerial vehicle comprises: an indication of regions of interest and, foran individual region of interest, a distance away from the region ofinterest that the unmanned aerial vehicle is to fly.
 6. The method ofclaim 1, wherein checking to determine whether the calculated predictedtrajectory of the unmanned aerial vehicle through the environment isfree of collisions comprises: comparing the three-dimensional model ofthe environment to the predicted trajectory of the unmanned aerialvehicle through the environment to determine whether the unmanned aerialvehicle would make contact with an object in the environment.
 7. Themethod of claim 1, further comprising: determining that the calculatedpredicted trajectory of the unmanned aerial vehicle through theenvironment is not free of collisions; and in response to determiningthat the calculated predicted trajectory of the unmanned aerial vehiclethrough the environment is not free of collisions, providing feedback tothe user that the calculated predicted trajectory of the unmanned aerialvehicle is not free of collisions.
 8. The method of claim 7, furthercomprising: in response to determining that the calculated predictedtrajectory of the unmanned aerial vehicle through the environment is notfree of collisions, identifying an interfering portion of theenvironment, and wherein providing feedback to the user that calculatedpredicted trajectory of the unmanned aerial vehicle is not free ofcollisions comprises displaying the interfering portion of theenvironment in a different color from non-interfering portions of theenvironment.
 9. The method of claim 7, further comprising: in responseto determining that the calculated predicted trajectory of the unmannedaerial vehicle through the environment is not free of collisions,modifying the information characterizing the visual representation ofthe three-dimensional model of the environment to prevent the display ofan area causing a collision.
 10. The method of claim 7, furthercomprising: in response to determining that the calculated predictedtrajectory of the unmanned aerial vehicle through the environment is notfree of collisions, modifying the proposed trajectory of the unmannedaerial vehicle to ensure that the unmanned aerial vehicle will follow apath that is free of collisions.
 11. The method of claim 1, wherein thereceived input related to the proposed flight path for the unmannedaerial vehicle through the environment includes proximity constraintsand a speed for the unmanned aerial vehicle.
 12. The method of claim 1,further comprising: generating a three-dimensional model of theenvironment based at least in part on information collected from thecamera, rangefinder, tilt encoder, and pan encoder at least in part by:capturing images of the environment with the camera; generating depthdata for objects within the captured images at least in part bydetermining or estimating the distance from the rangefinder to each of aplurality of the objects; and transforming the generated depth data toCartesian coordinates based at least in part on information collectedfrom the tilt encoder and the pan encoder.
 13. The method of claim 1,further comprising: causing the unmanned aerial vehicle to begin aflight in accordance with the predicted trajectory of the unmannedaerial vehicle through the environment; and tracking the flight of theunmanned aerial vehicle through the environment with a first scanner.14. The method of claim 13, wherein tracking the flight of the unmannedaerial vehicle through the environment comprises combining corner cubeinterferometry with output from a laser-based position sensing device togenerate position estimates for the unmanned aerial vehicle.
 15. Themethod of claim 13, further comprising: handing off responsibility fortracking the unmanned aerial vehicle from the first scanner to a secondscanner while the unmanned aerial vehicle is in flight.
 16. The methodof claim 13, further comprising: while tracking the flight of theunmanned aerial vehicle through the environment, comparing the flight ofthe unmanned aerial vehicle through the environment to the predictedtrajectory of the unmanned aerial vehicle through the environment. 17.The method of claim 16, further comprising: adjusting the flight of theunmanned aerial vehicle in response to determining, based on thecomparing, that the unmanned aerial vehicle has deviated from thepredicted trajectory of the unmanned aerial vehicle through theenvironment.
 18. A computer-readable storage medium storing instructionsthat, if executed by a computing system having a memory and a processor,cause the computing system to perform a method, the method comprising:providing, for presentation to a user, information characterizing avisual representation of a three-dimensional model of the environment,the environment including a plurality of objects; receiving, from theuser, input related to a proposed flight path for the unmanned aerialvehicle through the environment; calculating a predicted trajectory ofthe unmanned aerial vehicle through the environment based at least inpart on the received input and flight dynamics and capabilities of theunmanned aerial vehicle; and checking to determine whether thecalculated predicted trajectory of the unmanned aerial vehicle throughthe environment is free of collisions.
 19. The computer-readable mediumof claim 18, the method further comprising: based on an input from theuser, shifting a viewpoint from which the user views the visualrepresentation of the three-dimensional model of the environment.
 20. Asystem comprising: a camera; a rangefinder; a tilt encoder; a panencoder; a component configured to generate a three-dimensional model ofan environment based at least in part on information collected from thecamera, information collected from the rangefinder, informationcollected from the tilt encoder, and information collected from the panencoder; a component configured to provide, for presentation to a user,information characterizing a visual representation of thethree-dimensional model of the environment, the environment including aplurality of objects; a component configured to receive, from the user,input related to a proposed flight path for an unmanned aerial vehiclethrough the environment; and a component configured to calculate apredicted trajectory of the unmanned aerial vehicle through theenvironment based at least in part on the received input and flightdynamics and capabilities of the unmanned aerial vehicle.
 21. The systemof claim 20, further comprising: a component configured to check whetherthe calculated predicted trajectory of the unmanned aerial vehiclethrough the environment is free of collisions.
 22. The system of claim20, wherein information collected from the camera and informationcollected from the rangefinder comprise information collected from eachof a plurality of locations around the environment.